Introduction¶

In this notebook we will explore the extracted features from the WESAD dataset.

%reload_ext pretty_jupyter

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn import metrics

import sklearn.feature_selection as fs

import seaborn as sns
import plotly_express as px
import plotly.offline as pyo
import matplotlib.pyplot as plt

pd.set_option('display.max_columns', 100)
pd.set_option('display.max_rows', 100)
pyo.init_notebook_mode()

General Analysis¶

.tabset|O_O|.tabset-pills|O_O|.tabset-fade

First, we import the dataset.

data = pd.read_csv('../data/03_primary/WESAD/combined_subjects.csv')

We can observe that the all the data is numeric and there are no missing values. We will remove the first column as it is just a clone of the index.

data = data.drop([data.columns[0]], axis=1)

Dataset Column Overview¶

data.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 2091 entries, 0 to 2090
Data columns (total 44 columns):
 #   Column           Non-Null Count  Dtype  
---  ------           --------------  -----  
 0   net_acc_mean     2091 non-null   float64
 1   net_acc_std      2091 non-null   float64
 2   net_acc_min      2091 non-null   float64
 3   net_acc_max      2091 non-null   float64
 4   EDA_phasic_mean  2091 non-null   float64
 5   EDA_phasic_std   2091 non-null   float64
 6   EDA_phasic_min   2091 non-null   float64
 7   EDA_phasic_max   2091 non-null   float64
 8   EDA_smna_mean    2091 non-null   float64
 9   EDA_smna_std     2091 non-null   float64
 10  EDA_smna_min     2091 non-null   float64
 11  EDA_smna_max     2091 non-null   float64
 12  EDA_tonic_mean   2091 non-null   float64
 13  EDA_tonic_std    2091 non-null   float64
 14  EDA_tonic_min    2091 non-null   float64
 15  EDA_tonic_max    2091 non-null   float64
 16  BVP_mean         2091 non-null   float64
 17  BVP_std          2091 non-null   float64
 18  BVP_min          2091 non-null   float64
 19  BVP_max          2091 non-null   float64
 20  TEMP_mean        2091 non-null   float64
 21  TEMP_std         2091 non-null   float64
 22  TEMP_min         2091 non-null   float64
 23  TEMP_max         2091 non-null   float64
 24  ACC_x_mean       2091 non-null   float64
 25  ACC_x_std        2091 non-null   float64
 26  ACC_x_min        2091 non-null   float64
 27  ACC_x_max        2091 non-null   float64
 28  ACC_y_mean       2091 non-null   float64
 29  ACC_y_std        2091 non-null   float64
 30  ACC_y_min        2091 non-null   float64
 31  ACC_y_max        2091 non-null   float64
 32  ACC_z_mean       2091 non-null   float64
 33  ACC_z_std        2091 non-null   float64
 34  ACC_z_min        2091 non-null   float64
 35  ACC_z_max        2091 non-null   float64
 36  0_mean           2091 non-null   float64
 37  0_std            2091 non-null   float64
 38  0_min            2091 non-null   float64
 39  0_max            2091 non-null   float64
 40  BVP_peak_freq    2091 non-null   float64
 41  TEMP_slope       2091 non-null   float64
 42  subject          2091 non-null   int64  
 43  label            2091 non-null   int64  
dtypes: float64(42), int64(2)
memory usage: 718.9 KB

Dataset Shape¶

Descriptive Statistics¶

Now we will explore the data. We will start by looking at the distribution of the features.

Feature selection¶

.tabset|O_O|.tabset-pills|O_O|.tabset-fade

After loading and observing the dataset, it's time to find best features. Before doing this, we look at all the features available in the dataset.

data_ = data.copy()
cdf = pd.concat([data_.drop("label", axis=1), pd.get_dummies(data_["label"])], axis=1)
cdf.rename(columns={0: "amusement", 1: "baseline", 2: "stress"}, inplace=True)

corr = cdf.corr()
fig = px.imshow(corr[["amusement", "baseline", "stress"]], text_auto=True, color_continuous_scale=px.colors.sequential.Viridis)
fig = fig.update_layout(width=500, height=3800)
fig.show()

After looking at all features, we use SelectKBest from the sklearn library to find five best features.

# def get_best_features()
kBest = fs.SelectKBest(fs.f_classif, k=5)
res = kBest.fit_transform(data.drop(columns=['label']), data['label'])
filter = kBest.get_support()
df = pd.DataFrame(res, columns = data.columns[:-1][filter])
df = df.join(data['label'])
print(f"""Top features:\n{"  ".join(data.columns[:-1][filter])}""")

Top features:
net_acc_std  net_acc_max  EDA_tonic_mean  EDA_tonic_min  EDA_tonic_max

cdf = pd.concat([df.drop("label", axis=1), pd.get_dummies(df["label"])], axis=1)
cdf.rename(columns={0: "amusement", 1: "baseline", 2: "stress"}, inplace=True)

corr = cdf.corr()
fig = px.imshow(corr[["amusement", "baseline", "stress"]], text_auto=True, color_continuous_scale=px.colors.sequential.Viridis)
fig = fig.update_layout(width=500, height=800)
fig.show()

Data analysis by subject¶

.tabset|O_O|.tabset-pills|O_O|.tabset-fade In this section we plot top five features per subject to see if they will show the same pattern.

from plotly.subplots import make_subplots
import plotly.graph_objects as go

def plot_distribution(feature, nbr_cols=4):
    subjects = [2, 3, 4, 5, 6 ,7 ,8, 9, 10, 11, 13, 14, 15, 16, 17]
    titles = [f'Subject {x}' for x in subjects]
    plot = make_subplots(rows=len(subjects) // nbr_cols+1, cols=nbr_cols, subplot_titles=titles)

    row_n = 1
    col_n = 1
    for sub in subjects:
        csv = pd.read_csv(f'../data/03_primary/WESAD/subject_feats/S{sub}_feats_4.csv')
        plot.add_trace(go.Bar(y=csv[feature]), row_n, col_n, )
        col_n += 1
        if col_n > nbr_cols:
            col_n = 1
            row_n += 1

    plot.update_layout(height=1000)
    plot.show()

net_acc_std¶

plot_distribution('net_acc_std')

net_acc_max¶

plot_distribution('net_acc_max')

EDA_tonic_mean¶

plot_distribution('EDA_tonic_mean')

EDA_tonic_min¶

plot_distribution('EDA_tonic_min')

EDA_tonic_max¶

plot_distribution('EDA_tonic_max')

The effect of the stress level¶

.tabset|O_O|.tabset-pills|O_O|.tabset-fade Finally, we will compare all three states (baseline, stress and amusement) to see how the values differentiate between the states.

First, we prepare the combined dataset.

def get_label(label):
    frame = df.loc[df.label==label]
    frame.index = range(0, frame.shape[0])
    frame.index = pd.to_datetime(frame.index, unit='s')
    return frame

amusement = get_label(0)
baseline = get_label(1)
stress = get_label(2)

df.label = df.label.replace(0, 'amusement')
df.label = df.label.replace(1, 'baseline')
df.label = df.label.replace(2, 'stress')

Next, we define functions to plot data per state for each subject.

def split_states(data_, label_):
    print(data_.head())
    frame = data_.loc[data_.label==label_]
    frame.index = range(0, frame.shape[0])
    frame.index = pd.to_datetime(frame.index, unit='s')
    return frame

def plot_states(feature, nbr_cols=3):
    subjects = [2, 3, 4, 5, 6 ,7 ,8, 9, 10, 11, 13, 14, 15, 16, 17]
    titles = [f'Subject {x}' for x in subjects]
    plot = make_subplots(rows=5, cols=nbr_cols, subplot_titles=titles)

    row_n = 1
    col_n = 1
    for sub in subjects:
        csv = pd.read_csv(f'../data/03_primary/WESAD/subject_feats/S{sub}_feats_4.csv')

        csv['label'] = csv[['0','1','2']].idxmax(axis=1).astype('int32')
        plot.add_trace(go.Scatter(y=csv.loc[csv.label == 0][feature], line = {'color': '#00cc96'}, mode='lines', name = 'amusement'), row=row_n, col=col_n)
        plot.add_trace(go.Scatter(y=csv.loc[csv.label == 1][feature], line = {'color': '#ef553b'}, mode='lines', name = 'baseline'), row=row_n, col=col_n)
        plot.add_trace(go.Scatter(y=csv.loc[csv.label == 2][feature], line = {'color': '#19d3f3'}, mode='lines', name = 'stress'), row=row_n, col=col_n)

        col_n += 1
        if col_n > nbr_cols:
            col_n = 1
            row_n += 1

    tmp = 0
    for trace in plot['data']:
        tmp += 1
        if tmp > 3:
            trace['showlegend'] = False

    plot.update_layout(height=1500, title_text="Per Subject Data")
    plot.show()

net_acc_std¶

fig = px.line(df, y='net_acc_std', color='label', title="Combined Data")
fig.show()

After looking at the combined graph, let's see how it looks separately for each subject.

plot_states('net_acc_std')

net_acc_max¶

fig = px.line(df, y='net_acc_max', color='label', title="Combined Data")
fig.show()

After looking at the combined graph, let's see how it looks separately for each subject.

plot_states('net_acc_max')

EDA_tonic_mean¶

fig = px.line(df, y='EDA_tonic_mean', color='label', title="Combined Data")
fig.show()

After looking at the combined graph, let's see how it looks separately for each subject.

plot_states('EDA_tonic_mean')

EDA_tonic_min¶

fig = px.line(df, y='EDA_tonic_min', color='label', title="Combined Data")
fig.show()

After looking at the combined graph, let's see how it looks separately for each subject.

plot_states('EDA_tonic_min')

EDA_tonic_max¶

fig = px.line(df, y='EDA_tonic_max', color='label', title="Combined Data")
fig.show()

After looking at the combined graph, let's see how it looks separately for each subject.

plot_states('EDA_tonic_max')

Conclusion¶

After applying the sliding window to the dataset, we can conclude that the characteristics of the data did not change that much. Best correlating features are the same as in the initial 30s dataset and graphs of all best features show the same patterns.

From observing the graphs above we can conclude that for the EDA_tonic features for almost all the subjects, except for subject 17, have a noticeably higher and fluctuating values when they are stressed. Subject 17 anomaly could be explained by the fact that "the sensor was not fully attached throughout the entire duration of the study protocol", as it is stated in the dataset files.

In the case of the net_acc_std feature we can see that there are way more in the readings of stress patients and the values are generally higher, the difference in the values of stressed and baseline periods can be observed more clearly in the net_acc_max feature graphs as there aren't any big oscillations.

We can conclude that even a basic Machine Learning model, or a thresholding algorithm will be able to classify the stress levels with a high accuracy, because of the clear difference between the features of the different stress levels.